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ABSTBACT 

Fourteen criteria used to evaluate criterion 
referenced tests were assigned two sets of weights reflecting the 
characteristics of tests designed as a classroom resource or for 
program evaluation. Twenty eight currently available criterion 
referenced tests were rated against the criteria using each set of 
weights. A comparison of the scores obtained with each weighting 
system yielded significant differences. The findings of this study 
support the view that the same criteria cannot be used for all 
purposes and that therefore^ a criterion referenced test must be 
developed^ validated ^ and evaluated in terms of the purpose for which 
it is intended. (Author/BW) 
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It's the Metric That Counts 
or 

Criterion-Referenced Schizophrenia 

The academic and lay communities have, for over two decades, been try- 
ing to develop educational measurement systems which accurately gauge and 
monitor the performance by an individual or group of individuals on a given 
set of instructionaltasks, or behavioral objectives, in such a way as to 
yield pertinent and useable information to those persons providing the edu- 
cational experience. The results of these efforts are called criterion- 
referenced tests (CRTs), and they are deliberately constructed to permit 
score interpretations in terms of pre-specified performance standards. 
That is, CRTs can be used to describe what an individual br.si^oup of indi- 
viduals can or cannot do (Glaser and Nitko, 1971; Pophara and Husek, 1969). 

The reaction to CRTs has been enthusiastic from the start. Because they 
provide score interpretations in terms of the achievement of specific and 
measureable skills and behaviors, CRTs have had appeal to.those directly 
responsible for the education of students and the development and evaluation 
of educational programs. They also have had. appeal to teachers who found 
the results of standardized tests inadequate to assist them in planning 
lessons, and to many educators and psychologists who judged standardized, 
norm-referenced tests to be unfair and even' biased against individuals from 
under-privileged and minority groups. Finally, becausethe criterion- 
referenced approach was new, people have seen it as an opportunity to improve 
on some of the mistakes they perceived to be built into, or confused with, 
norm-referenced testing (Kosecoff and Fink, 1976b). 
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CRT's popularity and sanction by theoreticians -and practitioners has 
led to their frequent use for instructional diagnosis and placement and for 
measuring student achievement on educational tasks or objectives. In addi- 
tion, CRTs' are being suggested or used for other purposes like the eval- 
uation of educational programs and the National Assessment of Educational 
Progress (Wilson, 1974). In fact, many recently-issued requests for pro- 
posals from state and federal agencies to evaluate educational programs 
have specifically required prospective contractors to justify their selec- 
tion of standardized rather than CRT measures. 

The question then arises if the same CRTs can be used for classroom 
and evaluation purposes, and if not, what are the characteristics of CRTs 
best suited for each situation? Until recently, the answer to this question 
would probably have been yes, that the same CRTs can be effectively used 
for a variety of purposes. One reason for this answer is that publishers 
of CRTs have simultaneously recoranended their tests for use in diagnosing 
learning problems,. placing students in a curriculum, measuring progress, 
and evaluating instructional programs (Kosecoff, Klein, and Morgan, 1976). 
A second reason is that in the CRT literature, most educators and psycho- 
metricians have not distinguished between CRTs intended for different pur- 
poses either in terms of differences in physical characteristics or psycho- 
metric properties (Alkin et al. 1974). Third, by applying the same cri- 
teria and the same weights to all CRTs, efforts to evaluate CRTs also have 
not recognized different types of CRTs (Nafziger et al., 1975). 

Recently, some test developers, theoreticians and users have begun to 
stu(ly the characteristics of . CRTs designed for different purposes. For 
example, the Instructional Objectives Exchange is developing, for school 
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districts, customized tests that differ considerably in format and develop- 
ment from other coiimerci al ly available CRTs (Popham, 1976), and in a recent 
study for System Development Corporation (SDC), Kosecoff and Fink (1976a) 
concluded that no commercially available CRT was appropriate for use in 
SDC's longitudinal, nation-wide study of compensatory education. 

The purpose of this paper is to lend support to the view that different 
decision purposes require CRTs with different physical and theoretical pro- 
perties, and that therefore, CRTs must be evaluated in terms of the context 
in which they will be used. To do this, two situations in which CRTs are 
being used, the classroom resource and the program evaluation context, were 
considered and a sample of criteria that have been used or are being used to 
evaluate CRTs were assigned weights appropriate to each context. Then, 
twenty eight comnercially published CRTs were rated using each set of weights. 

The Classroom Resource Context 

One of the original and most prevalent uses of CRTs is as an instruc- 
tional aide to teachers in planning, evaluating, and individualizing instruc- 
tion. CRT results can be used by teachers to identify individual's or 
group's learning problems, to place students in a curriculum, to organize 
instructional groups, to monitor student's progress throughout a curriculum, 
and to measure the extent to which students mastered specific instructional 
objectives. In this context, CRTs can be seen primarily as tools used by 
teachers to obtain information for classroom management (Anatasi, 1968), 

The Program Evaluation Context 

CRTs can be used to provide information useful in evaluating educational 
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programs. This evaluation information can be used to modify and improve 
a still-developing program or to ascertain the effectiveness and worth of 
a mature program (Kosecoff and Fink, 1976b). In this context, CRTs can be 
seen primarily as tools used by evaluators to provide program developers 
and/or program sponsors with information about the nature and quality of 
a program's goals, outcomes, impacts, and costs. 

Evaluation Criteria 

In order to investigate differing characteristics of CRTs intended 
for more than one purpose, twenty eight currently available CRTs were 
reviewed using a dually-weighted set of fourteen criteria. The criteria 
chosen for the review were selected, based on a literature review, from 
criteria lists that are currently being used or have been used to evaluate 
educational tests (Hoepfner et al., 1970; Klein and Kosficoff, 1973; Kose- 
coff, Klein and Morgan, 1976; Kosecoff and Fink, 1976b; Buros, 1972). For 
this review, two sets of weights were assigned to each criteria, the first 
set of weights reflecting characteristics desirable of CRTs intended for 
use as a classroom resource and the secondfset of weights reflecting char- 
acteristics desirable of CRTs intended for use in program evaluation. 

An explanation of each criterion, its rating categories, and the 
assigned weights are:. 

(1) Overlap of ob.lectives across grade levels . Some or all of the 
test's objectives must be measured at each grade level. This criterion 
is particulary important in a program evaluation context^ order to make 
comparisons across grade levels or over time in terms of common educational 
objectives or skills. . 
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CRITERION 



RATING CATEGORIES 



WEIGHTS 
CLASSROOM PROGRAM 
RESOURCE EVALUATION 



Overlap of objectives 
across grade levels 



no overlap 0 

overlap in goals 1 

some overlap in objs.. 1 
overlap in goals, and 

some objectives 2 

total overlap T_ 



0 
2 
2 

3 

2 



(2) Grade level coverage . The number of grade levels for which some 
form of the CRT is appropriate. This criterion is slightly more important 
for CRTs intended for use in evaluations, since it is frequently necessary 
to interpret trends or outcomes in this context at more than one grade 
level. 



CRITERION 



RATING CATEGORIES 



WEIGHTS 
CLASSROOM PROGRAM 
RESOURCE EVALUATION 



Grade level coverage 



just one grade 
several grades 
(K-3 or 4-6) 
K-6 



0 
1 



0 
2 



(3) Number of test forms . The number of different tests or subtests 
(meas-uring different objectives) intended for a given grade level. Due to 
constraints related to test administration and the time available for test- 
ing in an evaluation context, it is desirable to have a limited number of 
test forms at each grade level. Just one test per grade level is preferred 
in order. to avoid problems with reliability that can arise when several 
test forms are combined. On the other hand, in a classroom context, many 
short test forms, each measuring just a few very specific instructional 
objectives might be preferred. 
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CRITERION 


RATING CATEGORIES 


WEIGHTS 
CLASSROOM 
RESOURCE 


PROGRAM 
EVALUATION 


Number of test forms 

per grade level 

(not alternate forms) 


just one test form 

2-9 tests 

10 or more tests 


0 
2 
3 


3 
1 
0 



(4) Special equipment needed for test administration . In view of the 
variation in equipment from school to school and in an effort to reduce 
costs-, test administration in an evaluation context should not require any 
special equipment (like cassettes or visual aids). In a classroom context 
it can be useful to make use of this equipment for example, in individual- 
izing testing. However, it is preferred that special equipment be available 



as an option and not be mandatory. 
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RATING CATEGORIES 


WEIGHTS 
CLASSROOM 
RESOURCE 


PROGRAM 
EVALUATION 


Special equipment for 
test administration 


none needed 
yes, but optional 
yes, mandatory 


2 
1 
0 


3 
1 

0 



(5a) Average time for administering a single test . (Criterion 5a 
applies just to CRTs with more than one test per grade level, excluding 
alternate test foms). The amount of time required, on the average, to 
administer a single test at a given grade level. For this criterion, the 
principle "the less testing time the better," applies to CRTs intended for 
use in either context, however, evaluations usually have more stringent 
limits on the total amount of time for testing. 
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CRITERION 


RATING CATEGORIES 


WEIGHTS 
CLASSROOM 
RESOURCE 


PROGRAM 
EVALUATION 


Average time for admin- 
istering a single test 
(if more than one test 
per level) 


10 or less minutes 

10-20 minutes 

20 or more minutes 


3 
2 
1 


1 

■ S 



(5b) Average time for administering all tests at a grade level (exclud- 
ing parallel forms). The amount of time needed to administer all tests 
(excluding parallel test forms) at a grade level. For use in an evaluation 
context, it is desirable that the CRT be designed to be completed within a 
class period. 



CRITERION 


RATING categories" 


WEIGHTS 
CLASSROOM 
RESOURCE 


PROGRAM 
EVALUATION 


Average time for test- 
ing for all forms at 
a grade level (not 
alternate forms) 


45 or less minutes 

45-90 minutes 

90 or more minutes 


0 
0 
1 


2 
1 
0 



(6) Mach i ne/s el f -s cori ng . This refers to whether the test can be 
machine scored, hand scored, or both. Sending CRTs to a publisher for 
machine-scoring, can waste valuable time in a classroom context, and con- 
sequently hand-score'able tests are prefe^^^ 

context where it is not infrequent to test thousands of students, machitie 
scoring is both efficient and accurate. 



> 
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CRITERION 


RATING CATEGORIES 


WEIGHTS 
CLASSROOM 
RESOURCE 


PROGRAM 
EVALUATION 


MjTchi ne/sel f scoring 


just machine 
just self 
both 


0 
3 
3 


3 — 

1 

3 



(7) Score interpretation . 'CRT scores are reported in terms of the 
level of performance achieved for each objective measured by a CRT. Some 
CRTs report scores as the number of items correctly answered per objective 
or as mastery of an objective, where mastery is an arbitrarily defined 
level of performance. CRTs using these score interpretation schemes usu- 
ally are intended for use in a classroom context and have very specific and 
operationally-defined objectives that can be directly measured by the test 
items. Other CRTs report scores in terms of true level of performance on 
an objective, referring to the portion of the total universe of items that 
could be correctly answered, or in terms of empirically-determined mastery 
levels. CRTs using this type of score interpretation scheme usually are 
intended for use in an evaluation context and have more generally-stated 
objectives for which it is difficult to assume that achievement of the 
test items necessarily reflects achievement of the larger objective. 



CRITERION 


RATING CATEGORIES 


WEIGHTS 
CLASSROOM 
RESOURCE 


PROGRAM 
EVALUATION 


Score interpretation 


# items per obj. or 
arbitrary mastery 

true score or empir- 
ically-based 
mastery 


2 

3 


0 ;■ ■•■^■-i. 
" 3 ■ ■ 
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(8) Curriculum match . Some CRTs are designed for use with a specific 
educational program (Baker, R.L.. 1972; Skager, 1973). CRTs that closely 
match a curriculum have objectives and test items that are associated 
with a particular set of educational materials and techniques. CRTs with 
a smaller match to a curriculum contain objectives and test items that 
are not necessarily associated with the specific skills or content of an 
educational program. However, they still may have been developed from 
several educational programs and consequently, have objectives and Items 
that reflect the bias inherent in these programs. Conversely, CRTs with 
no match to a curriculum are based on objectives and test items that are 
Independent of any educational program, and, therefore, can be used to 
compare several different educational programs. In the classroom context 
it is useful to use a CRT that is closely matched to a curriculum, while 
in an evaluation context involving comparisons, the opposite It true. 



CRITERION 


RATING CATEGORIES 


WEIGHTS 
CLASSROOM 
RESOURCE 


PROGRAM 
EVALUATION 


Curriculum match 



none 

somewhat 
close 


2 

3- ■ 


3 
2 

1 • 



(9) Who can interpret scores. This criterion refers to whether a 
teacher can interpret the CRT^s score or if an expert is needed. Clearly, 
In the classroom context a teacher should be able to Interpret scores, 
while the evaluator is, by training, skilled in, test interpretation^ 
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CRITERION 


RATING CATEGORIES 


WEIGHTS 
CLASSROOM 
RESOURCE 


PROGRAM 
EVALUATION 


Who can Interpret 
scores 


teacher 
specialist 


3 
0 


1 
0 



(10) Formal field test .— It Is a well excepted fact that any test 
should be carefully validated. However, a carefully documented field test 
report Is particularly Important In a. program evaluation context because 
far-reaching decisions Iriay be made on the basis of the evaluation findings. 



WEIGHTS 

/.nT-rrnrnM DATTWP rUTCPnDTrC CLASSROOM ' PROGRAM 

CRITERION RATING CATEGORIES RESOURCE EVALUATION 



Formal field test none mentioned 0 0 

mention but little 0 0 

documentation 

yes, restricted scope .1 .2 

yes, national scope 2 3 



(11) Stability/number of Items per objective . A determination of the 
number of Items that must be tested In order to obtain a stable score on 
an Objective. For a CRT used In a classroom context. It Is necessary to 
establish stability for Individual examinees, while for a CRT used In an 
evaluation context. It Is desirable to determine stability for groups of 
examinees as well as Individuals. (Note, fewer Items are probably needed to 
obtain stable scores for a group than for an Individual.) 



CRITERION 


RATING CATEGORIES. 


WEIGHTS 
CLASSROOM' 
RESOURCE 


PROGRAM 
EVALUATION 


Stability/number of 
Items per objective 


stability for Indlvs. 
stability for groups 
both ; 


, ■2 ^ 


.2 
2 
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(12) Sensitivity to instruction . This criterion refers to how well 
a CRT distinguishes between those who have and those who have not bene- 
fited from instruction. In a classroom context, a teacher is most likely 
interested in using a CRT to determine if students mastered one or more • 
instructional objectives. It is important therefore, that the CRT be. 



sensitive to the teacher 


's instruction. 




CRITERION 


RATING CATEGORIES 


WEIGHTS 
CLASSROOM PROGRAM 
RESOURCE EVALUATION 


Sensitivity to 
instruction 


study not documented 
study documented 


0 0 
2 1 



(13) Subject area comprehensiveness . Some CRTs attempt to cover a 
given subject area by measuring many specific, objectives. Other CRTs 
attempt measuring just a limited number of objectives. However, the 
objectives are selected so that performance on them is general izable to 
other skills associated with the subject area. In an evaluation context, 
it is desirable to use the secbM"k"iFd~ofTRFTr^o 
able amount of information and to meet testing time constraints. In a 
classroom context, the first type of CRT is probably more useful since 
students can be tested on instructional objectives as they are introduced. 



CRITERION 


RATING CATEGORIES 


. WEIGHTS 
CLASSROOM PROGRAM j 
RESOURCE EVALUATION;- 


Subject area 
comprehensiveness 


test measures mores 
than 20 objectives 

test salnp^es~objsr-*-"' 
but samplie has prer 
dictive validil^y 


; ■ ■ , 

2 \,:' - ■ ■ .1. - \;{rr^:^:^ 
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(14) Availability of comparative information . In addition to a 
CRT interpretation, norm- referenced interpretations can also be provided. 
Comparative data describing how other students have performed on the same 
objective are useful in both the classroom resource and program evaluation 
contexts to enhance the meaningfulness of CRT scores. 



CRITERION RATING CATEGORIES 


WEIGHTS 
CLASSROOM 
RESOURCE- ■ 


'program 
evaluation 


Availability of com- available 
parative information not available 


2 

0 . 


2 
6 


A complete rating scale can be found in Figure 1. 



see page 13 for Figure 1 



Results of the Review 
Twenty eight CRTs currently being marketed by major publishers of 
educational tests were reviewed. The names of the systems reviewed and 
their respective publishers are listed in Table 1. 

• see page 14 for Table 1 

Copies of the CRTs were obtained from the Center for the Study of' Evaluation's 
test library. Each CRT was independently reviewed twice using the criteria 
selected for this purpose arid discrepancies were resolved byNthe reviewers. 
Any remaining questions, that is, those resulting from unclear or insuffi- 
cient information were followed-up by a phone call to the publisher. 

For the review, when sufficient information could not be obtained to . . 
rate a criterion, a score of zero was assigned. Review scores were computed v 
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Figure 1: Rating Categories 





CRITERIA 


RMIfJR CATEGORIES* 


POINTS 
CLASSROOM 


.. POlms 

FVA1.UATI0N • 


• 


• 1. Overlap of objectives 
across grade levels 

• 


no overlap 

overlap in goals 

some overlap in objectives 

overlap in goals and some 

objectives 

total ovorlao 


0 

1 
1 

2 
1 


0 

z 

2 

3 
2 




2. Grade level coverage 


just 1 grade 

several grades (K-3 6t 4-6) 
K-6 


0 
1 

2 -• 


0 

'• 2 ••• 
.. . 3 




3,. Number of test forms 
per grade level 
^nnt altprnat© forms ^ 


just 1 test form 
• 2-9 tests • J 
10 or more tests 


0 • 

2 

3 


3 
1 
0 




4. Special equipment for 
• test administration 


none needed . 
yes, but optional 
yes, mandatory 


2 
1 
0 


3 
1 
0 




5a. Average time for admin- 
. Istering a single test 
(If more than one test 
form per level) 


< 10 min. 
10-20 min 
>20 min 

not applicable 


.. 3 
2 
1 
0 


.. 1 
0 

. 0 
0 




5b. Average time for admin- 
istering all tests at 
a grade level (excluding 
parallel test forms) 


<45 min. 

T5-90 min. ^. . 
>90 min 


0 
0 

1 


2 
1 
0 




6. Machine/Self 
scoring 


Just machine 
just self 
both 


0 
3 
3 


3 

1 . . 
3 - 




7. Score Interpretation 


# Items ;per obj. or 
arbitrary mastery 

true score or empir- 
ically-based mastery 


2 
3 


0 
3 




3^ Curriculum fnatch 


none 
somewhat 


1 

— 2-- — 


3 

1 






close 


3 




9. Who can. Interpret 
scores 


teacher . « 
specialist 


3 
0 


1 

.0 




10. Formal field test 


none mentioned 

mention but little documen- 

"tfltlon 

yes, restricted scope 
yes, national scope 


•0 
0 

1 

2 


0 

0 

2 
3 




11. Stability-number 

of Items per objective 


stability for Individuals 
stability for groups 
both 


2 
1 
2 


2 
2 
. 2 




12. Sensitivity to in- 
struction 


study not documented 
study documented 


b 

2 


0 
1 




Aw* wuvjwVb— oi wo 

coiqirchcnslvcncss ' 


''test measures many 
Qb^fec^fves ) " 
test Isampl es obJect1y«$ ^ 
but saniple. has predictive 
validity ^ 


2 
\ 


2 




14. Avanabllity of com- 
. paratlve Information 


available 

not V available ; 


• 2 • 
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Table 1 



Alphabetical listing of publisher and test systems reviewed 



American Guidance Service 

Key Math - Diagnostic Arlthmatic Test 
Woodcock Reading Mastery 

American Testing Company 

Mathematics Inventory Probe 
Reading Inventory Probe I 

aB/McGraw-Hin 

Comprehensive Tests cf Basic Sk1\ls (CTBS/S) - Mathematics 
Comprehensive Tests of Basic Skills (CTBs/S) - Reading 
Diagnostic Mathematics Inventory 
Dbjectlves-Referenced I3ank of Items and tests (DRBIT) 
Prescriptive Reading Inventory 

Educational and Industrial Testing Service 

Tests of Achievement In Basic Skills (TABS) - Mathematics 
Tests of Achievement In Basic Skills (TABS) - Reading 

Educational Development Corporation 

Individualized Criterion-Referenced Testing - Mathematics 
Individualized Criterion-Referenced Testing - Reading 

Harcourt Brace Jovanovlch 

SkllU Monitoring System - Reading (not yet available) 
1973 Stanford Mathematics Tests 
1973 Stanford Reading Tests 

Houghton - Mifflin 

Individual Pupil Monitoring System - Mathematics 
Individual Pupil Monitoring System - Reading 

Instructional Objectives Exchange* 

Objectives Based Test Sets - Hathematlci 
Objectives Based Test Sets - Reading 

National Evaluation Systems 

Comprehensive Achievement Monitoring (CAM) - Mathematics 
Comprehensive Achievement Monitoring (CAM) - Reading 



Richard Zweig Associates. Inc. 

Fountain Valley Teacher Support System - Mathematics 
Fountain Valley Teacher Support System - Reading 

Scholastic Testing Service 

Analysis of Skills - Mathematics 
Analysis of Skills - Reading 

Science Research Associates 

Mastery: An Evaluation Tool - Mathematics 
Mastery: An Evaluation TooT - SOBAR 



Metric 
15 

as the proportion of possible points earned by a CRT and separate scores 
for the two weighting systems were reported for each CRT. The rating 
scores obtained for each CRT are reported in Table 2. In order to main- 
tain publisher anonymity, the ratings are listed in random order and do 
not correspond with the alphabetical listing of publishers found in Table 
1. 

see page 17 for Table 2 

As can be seen from Table 2, there Is considerable variation in the 
scores each CRT earned using the two different weighting scales. When 
rated within the classroom resource context, the highest percentage ob- 
tained was 80 percent, with a low of 43 percent. Within the program eval- 
uation context, the spread in the range of high and "Tow percentages was 
quite similar, being 83 percent and 43 percent, respectively. A tentative 
conclusion, based on a review of the percentages earned by a CRT In these 
two contexts, might be that in trying to achieve both the classroom 
resource and program evaluation functions, CRTs are, for the most part, 
only marginally fulfilling each purpose. 

Concl us ions 

In this study fourteen criteria used to evaluate CRTs were assigned 
two sets of weights reflecting the characteristics of CRTs designed as a 
classroom resource or for program evaluation. Currently available CRTs 
were rated against the criteria using each set of weights. A comparison 
of the scores obtained with each weighting system yielded significant 
differences. The findings of this studly support the view that the same 
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criteria cannot be used for all purposes and that therefore, a CRT must 
be developed, validated, and evaluated in terms of the purpose for which 
i t is intended. 
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Table 2 



SCORE (expressed in percents) 
CLASSROOM RESOURCE PROGRAM EVALUATION 


65 % 


43 % 


65 


43 


74 


66 


51 


74 


80 


80 ' 


77 


83 


57 


54 


66 


54 


71 


80 


63 


80 


57 


"54 


57 


51 


77 


71 


60 


63 . 


51 


77 


54 


77 


49 


51 


51 


49 


46 


51 


49 


49 


43 


51 


43 


51 


54 


74 


54 


-74 


57 


69 


49 


69 


49 


66 


43 


57 
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